Nonlinear adaptive filtering with FIR synapses and adaptive activation functions

This paper focuses on multilayer perceptron neural networks where the activation functions are adaptive and where each neuron synapse is modelled by a finite impulse response (FIR) filter. A simplified architecture consisting of a variable activation (VA) function which is sandwiched between two FIR synapses is studied. The VA function consists of a mixed linear-tanh sigmoid with a parameter which controls the linear region.The VA parameters and FIR synaptic weights are updated using a modified form of the instantaneous-cost (IC) temporal backpropagation algorithm [1]. Simulations for identifying cascaded nonlinear transfer functions with internal memory and arbitrary activation functions illustrate the improved modelling performance over models with non-adaptive activation functions.